Feeling for Sound:Mapping Sonic Data to Haptic Perceptions

This paper presents a system for exploring different dimensions of a soundthrough the use of haptic feedback. The Novint Falcon force feedback interfaceis used to scan through soundfiles as a subject moves their hand horizontallyfrom left to right, and to relay information about volume, frequency content,noisiness, or potentially any analysable parameter back to the subject throughforces acting on their hand. General practicalities of mapping sonic elements to physical forces areconsidered, such as the problem of representing detailed data through vaguephysical sensation, approaches to applying forces to the hand that do notinterfering with the smooth operation of the device, and the relative merits ofdiscreet and continuous mappings. Three approaches to generating the forcevector are discussed: 1) the use of simulated detents to identify areas of anaudio parameter over a certain threshold, 2) applying friction proportional tothe level of the audio parameter along the axis of movement, and 3) creatingforces perpendicular to the subject's hand movements.Presentation of audio information in this manner could be beneficial for`pre-feeling' as a method for selecting material to play during a liveperformance, assisting visually impaired audio engineers, and as a generalaugmentation of standard audio editing environments

Nonlinear Dynamics In Musical Interactions

This thesis examines nonlinear dynamical processes in musical tools, identifying certain roles that they play in creative interactions with existing tools, and investigates the roles they might play in digital tools. Nonlinear dynamical processes are fundamental in the everyday physical world. They lie at the core of many acoustic instruments, playing a particularly significant role in bowed and blown instruments. Two major studies are presented that approach these issues from different perspectives. Firstly a set of comparative studies explore the ways in which musicians engage with systems that do and do not incorporate nonlinear dynamical processes. Secondly, interviews with a range of musicians engaged in contemporary musical practices — particularly free improvisation — are used to investigate the role of nonlinear dynamical processes in instrumental interactions in relation to unpredictability and creative exploration. Evidence is presented demonstrating that nonlinear dynamical processes can be drawn on as resources for exploration over long time periods. An approach to creative interaction that explicitly draws on the properties of nonlinear dynamical processes is uncovered and connected to material-oriented notions of creative processes. Nonlinear dynamics are shown to facilitate a productive ‘‘sweet spot’’ between unpredictability and complexity on the one hand, and detailed, sensitive, deterministic control, coupled with the potential to repeat and develop particular actions on the other. The importance of timing in interactions with nonlinear dynamical processes is highlighted as being significant in creating explorable interactions, particularly close to critical thresholds. A distinction is raised between instantaneous unpredictabilities that emerge from the interaction with the tool (interactional ), and unpredictabilities that result from the unexpected implications of the conjunction of otherwise anticipated elements (combinatorial). While the usefulness of the latter in creative interactions is frequently acknowledged in HCI research, the former is often excluded, or seen as a hinderance or obstruction. Engagements with nonlinear dynamical processes in existing musical instruments and practices provide clear evidence of the utility of both nonlinear dynamics, and interactional surprises more generally, suggesting that they can be of use in other domains where creative exploration is a concern

Dynamical Interactions with Electronic Instruments

This paper examines electronic instruments that incorporate dynamical systems, where the behaviour of the instrument depends not only upon the immediate input to the instrument, but also on the past input. Five instruments are presented as case studies: Michel Waisvisz’ Crackle-box, Dylan Menzies’ Spiro, no-input mixing desk, the author’s Feedback Joypad, and microphone-loudspeaker feedback. Links are suggested between the sonic affordances of each instrument and the dynamical mechanisms embedded in them. These affordances are contrasted with those of non-dynamical instruments such as the Theremin and sample-based instruments. This is discussed in the context of contemporary, material-oriented approaches to composition and particularly to free improvisation where elements such as unpredictability and instability are often of interest, and the process of exploration and discovery is an important part of the practice

Musical pathways through the no-input mixer

This paper examines the use of the no-input mixing desk—or feedback mixer—across a range of musical practices. The research draws on twenty two artist interviews conducted by the authors, and on magazine and forum archives. We focus particularly on how the properties of the no-input mixer connect with the musical, aesthetic and practical concerns of these practices. The affordability, accessibility, and non-hierarchical nature of the instrument are examined as factors that help the idea spread, and that can be important political dimensions for artists.The material, social and cultural aspects are brought together to provide a detailed picture of the instrument that goes beyond technical description. This provides a useful case study for NIME in thinking through these intercon- nections, particularly in looking outwards to how musical instruments and associated musical ideas travel, and how they can effect change and be changed themselves in their encounters with real-world musical contexts

Nonlinear Dynamical Processes in Musical Interactions: investigating the role of nonlinear dynamics in supporting surprise and exploration in interactions with digital musical instruments

Nonlinear dynamical processes play a central role in many acoustic instruments, yet they rarely feature in digital instruments, and are little understood from an interaction design perspective. Such processes exhibit behaviours that are complex, time-dependent, and chaotic, yet in the context of acoustic instruments can facilitate interactions that are explorable, learnable and repeatable. This suggests that these processes merit deeper investigation for digital music interaction design. Two studies are presented which investigate user interaction with nonlinear dynamical musical tools. A lab-based study used four purpose-built digital musical instruments to test interaction designs featuring nonlinear dynamical processes. Evaluations with 28 musicians demonstrated the potential for these processes to provoke creative surprises, and support exploration without a corresponding loss of control. A subsequent ethnographically-informed study with 24 musicians linked these findings to a mode of engagement which we term ‘edge-like interaction’. Edge-like interactions draw on the complex, unpredictable behaviours found in nonlinear dynamical processes close to critical thresholds, facilitating creative exploration. The two complementary studies provide evidence both for the existing importance of nonlinear dynamical processes in musical interactions with acoustic interactions, and their potential for deployment in the development of new creative digital technologies, musical or otherwise

The Role of Nonlinear Dynamics in Musicians' Interactions with Digital and Acoustic Musical Instruments

Nonlinear dynamical processes are fundamental to the behaviour of acoustic musical instruments, as is well explored in the case of sound production. However, such processes may have profound and under-explored implications for how musicians interact with instruments. While nonlinear dynamical processes are ubiquitous in acoustic instruments, they are present in digital musical tools only if explicitly implemented. Thus, an important resource with potentially major effects on how musicians interact with acoustic instruments is typically absent in the way musicians interact with digital instruments. 24 interviews with free improvising musicians were conducted to explore the role that nonlinear dynamics play in the participants’ musical practices, and to understand how such processes can afford distinctive methods of creative exploration. Thematic analysis of the interview data is used to demonstrate the potential for nonlinear dynamical processes to provide repeatable, learnable, controllable and explorable interactions, and to establish a vocabulary for exploring nonlinear dynamical interactions. Two related approaches to engaging with nonlinear dynamical behaviours are elaborated: edge-like interaction which involves the creative use of critical thresholds; and deep exploration which involves exploring the virtually unlimited subtleties of a very small control region. The elaboration of these approaches provides an important bridge that connects the concrete descriptions of interaction in musical practices on the one hand, to the more abstract mathematical formulation of nonlinear dynamical systems on the other

Nonlinear Dynamical Systems as Enablers of Exploratory Engagement with Musical Instruments

This paper presents a small scale study that examined links between the inclusion of nonlinear dynamical processes in musical tools and particular kinds of engagement. Communication-oriented attitudes to engagement that view the tool as a medium for transmission of ideas are contrasted with material-oriented attitudes that focus on the specific sonic properties and behaviours of a given tool, and the latter are linked to the inclusion of nonlinear dynamical elements. Methodological issues are raised and discussed, particularly with regard to the holistic nature of musical instruments, the difficulties of independently testing isolated design elements, and potential methods for addressing these difficulties

Dynamical Systems in Interaction Design for Improvisation

This paper proposes the use of, and investigation of the value of, nonlinear dynamical elements in mappings between human input and system output in interactive systems. Motivation for this, and a case study, are drawn from the practices of free, aural improvisers in digital and acoustic music. Nonlinear dynamical systems in existing sound creation mechanisms help create the rich affordances of many acoustic instruments, notably reed instruments. Dynamical systems also play a key role in electronic instruments, with many performers placing the exploration of feedback processes at the centre of their practice. We propose that the use of nonlinear dynamical elements can be usefully moved up from output mechanisms and incorporated explicitly at a higher level in the mappings between human input and system output in digital music systems. However, digital music is not the only area of human activity where divergent, open-ended, exploratory thinking is valued. We thus propose the incorporation of, and investigation of the value of, nonlinear dynamical elements in mappings between input and output in interactive systems more generally, in particular when designing for domains where divergent problem solving and problem seeking play an important role

Investigating the effects of introducing nonlinear dynamical processes into digital musical interfaces

This paper presents the results of a study that explores the effects of including nonlinear dynamical processes in the design of digital musical interfaces. Participants of varying musical backgrounds engaged with a range of representative systems, and their behaviours, responses and attitudes were recorded and analysed. The study suggests links between the inclusion of such processes and the affordance of exploration and serendipitous discovery. Relationships between musical instruments and nonlinear dynamics are discussed more broadly, in the context of both acoustic and electronic musical tools. Links between the properties of nonlinear dynamical systems and the priorities of experimental musicians are highlighted and related to the findings of the study